{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "8ae67518",
   "metadata": {},
   "source": [
    "# Enrich Standard Geographies\n",
    "\n",
    "Standard geographies are jurisdictional areas determined by government agencies. At the highest level these are the countries of the world. Within these countries the hierarchical levels have different names. If you already have a list of jurisdictional area identifiers such as postal (ZIP) codes or US Census Block Group Identifiers, these can be used directly as input to the `enrich` method to retrieve demographic information about these jurisdictional areas for analysis."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "a0fda81e",
   "metadata": {},
   "source": [
    "## Example Use Case - Variable Variance\n",
    "\n",
    "Just as before, we are going to retrieve high variance variables, but this time we are going to look up the unique identifiers for all the US Census Block Groups in Seattle."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "cbaa0921",
   "metadata": {},
   "source": [
    "### Create a Country\n",
    "\n",
    "Our analysis starts with identifying the country we are going to be working with and instantating an `arcgis.geoenrichment.Country` object referencing an `arcgis.gis.GIS` source to use for analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "926b3779",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Country - United States (GIS @ https://geosaurus.maps.arcgis.com version:10.3)>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from arcgis.geoenrichment import Country\n",
    "from arcgis.gis import GIS\n",
    "\n",
    "gis = GIS(profile=\"your_online_profile\")\n",
    "usa = Country(\"usa\", gis=gis)\n",
    "\n",
    "usa"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "0a3280d9",
   "metadata": {},
   "source": [
    "### Selecting Data to Start\n",
    "\n",
    "Next, we are using Pandas DataFrame filtering to identify a subset of variables to focus on from the thousands available."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "a873eca7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>alias</th>\n",
       "      <th>data_collection</th>\n",
       "      <th>enrich_name</th>\n",
       "      <th>enrich_field_name</th>\n",
       "      <th>description</th>\n",
       "      <th>vintage</th>\n",
       "      <th>units</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>NOHS_CY</td>\n",
       "      <td>2022 Pop Age 25+: &lt; 9th Grade</td>\n",
       "      <td>educationalattainment</td>\n",
       "      <td>educationalattainment.NOHS_CY</td>\n",
       "      <td>educationalattainment_NOHS_CY</td>\n",
       "      <td>2022 Population Age 25+: Less than 9th Grade (...</td>\n",
       "      <td>2022</td>\n",
       "      <td>count</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>SOMEHS_CY</td>\n",
       "      <td>2022 Pop Age 25+: High School/No Diploma</td>\n",
       "      <td>educationalattainment</td>\n",
       "      <td>educationalattainment.SOMEHS_CY</td>\n",
       "      <td>educationalattainment_SOMEHS_CY</td>\n",
       "      <td>2022 Population Age 25+: 9-12th Grade/No Diplo...</td>\n",
       "      <td>2022</td>\n",
       "      <td>count</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>HSGRAD_CY</td>\n",
       "      <td>2022 Pop Age 25+: High School Diploma</td>\n",
       "      <td>educationalattainment</td>\n",
       "      <td>educationalattainment.HSGRAD_CY</td>\n",
       "      <td>educationalattainment_HSGRAD_CY</td>\n",
       "      <td>2022 Population Age 25+: High School Diploma (...</td>\n",
       "      <td>2022</td>\n",
       "      <td>count</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>GED_CY</td>\n",
       "      <td>2022 Pop Age 25+: GED</td>\n",
       "      <td>educationalattainment</td>\n",
       "      <td>educationalattainment.GED_CY</td>\n",
       "      <td>educationalattainment_GED_CY</td>\n",
       "      <td>2022 Population Age 25+: GED/Alternative Crede...</td>\n",
       "      <td>2022</td>\n",
       "      <td>count</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>SMCOLL_CY</td>\n",
       "      <td>2022 Pop Age 25+: Some College/No Degree</td>\n",
       "      <td>educationalattainment</td>\n",
       "      <td>educationalattainment.SMCOLL_CY</td>\n",
       "      <td>educationalattainment_SMCOLL_CY</td>\n",
       "      <td>2022 Population Age 25+: Some College/No Degre...</td>\n",
       "      <td>2022</td>\n",
       "      <td>count</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>92</th>\n",
       "      <td>VAL1M_CY</td>\n",
       "      <td>2022 Home Value $1 Million-1499999</td>\n",
       "      <td>Wealth</td>\n",
       "      <td>Wealth.VAL1M_CY</td>\n",
       "      <td>Wealth_VAL1M_CY</td>\n",
       "      <td>2022 Home Value $1,000,000-$1,499,999 (Esri)</td>\n",
       "      <td>2022</td>\n",
       "      <td>count</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>93</th>\n",
       "      <td>MEDVAL_CY</td>\n",
       "      <td>2022 Median Home Value</td>\n",
       "      <td>Wealth</td>\n",
       "      <td>Wealth.MEDVAL_CY</td>\n",
       "      <td>Wealth_MEDVAL_CY</td>\n",
       "      <td>2022 Median Home Value (Esri)</td>\n",
       "      <td>2022</td>\n",
       "      <td>currency</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>94</th>\n",
       "      <td>AVGVAL_CY</td>\n",
       "      <td>2022 Average Home Value</td>\n",
       "      <td>Wealth</td>\n",
       "      <td>Wealth.AVGVAL_CY</td>\n",
       "      <td>Wealth_AVGVAL_CY</td>\n",
       "      <td>2022 Average Home Value (Esri)</td>\n",
       "      <td>2022</td>\n",
       "      <td>currency</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95</th>\n",
       "      <td>VALBASE_CY</td>\n",
       "      <td>2022 Home Value Base</td>\n",
       "      <td>Wealth</td>\n",
       "      <td>Wealth.VALBASE_CY</td>\n",
       "      <td>Wealth_VALBASE_CY</td>\n",
       "      <td>2022 Owner Occupied Housing Units by Value Bas...</td>\n",
       "      <td>2022</td>\n",
       "      <td>count</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>96</th>\n",
       "      <td>WLTHINDXCY</td>\n",
       "      <td>2022 Wealth Index</td>\n",
       "      <td>Wealth</td>\n",
       "      <td>Wealth.WLTHINDXCY</td>\n",
       "      <td>Wealth_WLTHINDXCY</td>\n",
       "      <td>2022 Wealth Index (Esri)</td>\n",
       "      <td>2022</td>\n",
       "      <td>count</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>97 rows × 8 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "          name                                     alias  \\\n",
       "0      NOHS_CY             2022 Pop Age 25+: < 9th Grade   \n",
       "1    SOMEHS_CY  2022 Pop Age 25+: High School/No Diploma   \n",
       "2    HSGRAD_CY     2022 Pop Age 25+: High School Diploma   \n",
       "3       GED_CY                     2022 Pop Age 25+: GED   \n",
       "4    SMCOLL_CY  2022 Pop Age 25+: Some College/No Degree   \n",
       "..         ...                                       ...   \n",
       "92    VAL1M_CY        2022 Home Value $1 Million-1499999   \n",
       "93   MEDVAL_CY                    2022 Median Home Value   \n",
       "94   AVGVAL_CY                   2022 Average Home Value   \n",
       "95  VALBASE_CY                      2022 Home Value Base   \n",
       "96  WLTHINDXCY                         2022 Wealth Index   \n",
       "\n",
       "          data_collection                      enrich_name  \\\n",
       "0   educationalattainment    educationalattainment.NOHS_CY   \n",
       "1   educationalattainment  educationalattainment.SOMEHS_CY   \n",
       "2   educationalattainment  educationalattainment.HSGRAD_CY   \n",
       "3   educationalattainment     educationalattainment.GED_CY   \n",
       "4   educationalattainment  educationalattainment.SMCOLL_CY   \n",
       "..                    ...                              ...   \n",
       "92                 Wealth                  Wealth.VAL1M_CY   \n",
       "93                 Wealth                 Wealth.MEDVAL_CY   \n",
       "94                 Wealth                 Wealth.AVGVAL_CY   \n",
       "95                 Wealth                Wealth.VALBASE_CY   \n",
       "96                 Wealth                Wealth.WLTHINDXCY   \n",
       "\n",
       "                  enrich_field_name  \\\n",
       "0     educationalattainment_NOHS_CY   \n",
       "1   educationalattainment_SOMEHS_CY   \n",
       "2   educationalattainment_HSGRAD_CY   \n",
       "3      educationalattainment_GED_CY   \n",
       "4   educationalattainment_SMCOLL_CY   \n",
       "..                              ...   \n",
       "92                  Wealth_VAL1M_CY   \n",
       "93                 Wealth_MEDVAL_CY   \n",
       "94                 Wealth_AVGVAL_CY   \n",
       "95                Wealth_VALBASE_CY   \n",
       "96                Wealth_WLTHINDXCY   \n",
       "\n",
       "                                          description vintage     units  \n",
       "0   2022 Population Age 25+: Less than 9th Grade (...    2022     count  \n",
       "1   2022 Population Age 25+: 9-12th Grade/No Diplo...    2022     count  \n",
       "2   2022 Population Age 25+: High School Diploma (...    2022     count  \n",
       "3   2022 Population Age 25+: GED/Alternative Crede...    2022     count  \n",
       "4   2022 Population Age 25+: Some College/No Degre...    2022     count  \n",
       "..                                                ...     ...       ...  \n",
       "92       2022 Home Value $1,000,000-$1,499,999 (Esri)    2022     count  \n",
       "93                      2022 Median Home Value (Esri)    2022  currency  \n",
       "94                     2022 Average Home Value (Esri)    2022  currency  \n",
       "95  2022 Owner Occupied Housing Units by Value Bas...    2022     count  \n",
       "96                           2022 Wealth Index (Esri)    2022     count  \n",
       "\n",
       "[97 rows x 8 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "enrich_vars = (\n",
    "    usa.enrich_variables[\n",
    "        (usa.enrich_variables.name.str.lower().str.contains(\"cy\"))\n",
    "        & (\n",
    "            (usa.enrich_variables.data_collection == \"occupation\")\n",
    "            | (usa.enrich_variables.data_collection == \"Wealth\")\n",
    "            | (usa.enrich_variables.data_collection == \"financial\")\n",
    "            | (usa.enrich_variables.data_collection == \"educationalattainment\")\n",
    "            | (usa.enrich_variables.data_collection == \"language\")\n",
    "            | (usa.enrich_variables.data_collection == \"healthinsurancecoverage\")\n",
    "            | (usa.enrich_variables.data_collection == \"veterans\")\n",
    "            | (usa.enrich_variables.data_collection == \"yearmovedin\")\n",
    "            | (usa.enrich_variables.data_collection == \"yearbuilt\")\n",
    "            | (usa.enrich_variables.data_collection == \"population\")\n",
    "            | (usa.enrich_variables.data_collection == \"occupation\")\n",
    "            | (usa.enrich_variables.data_collection == \"housingcosts\")\n",
    "        )\n",
    "    ]\n",
    "    .drop_duplicates(\"name\")\n",
    "    .reset_index(drop=True)\n",
    ")\n",
    "\n",
    "enrich_vars"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "3555dfe3",
   "metadata": {},
   "source": [
    "### Get the Geographic Level\n",
    "\n",
    "Here, we are retrieving `levels` and using the `level_name` colum to discover valid values for the `enrich` method's `standard_geography_level` parameter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "28ad9f82",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>level_name</th>\n",
       "      <th>singular_name</th>\n",
       "      <th>plural_name</th>\n",
       "      <th>alias</th>\n",
       "      <th>level_id</th>\n",
       "      <th>admin_level</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>block_groups</td>\n",
       "      <td>Block Group</td>\n",
       "      <td>Block Groups</td>\n",
       "      <td>Block Groups</td>\n",
       "      <td>US.BlockGroups</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>tracts</td>\n",
       "      <td>Census Tract</td>\n",
       "      <td>Census Tracts</td>\n",
       "      <td>Census Tracts</td>\n",
       "      <td>US.Tracts</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>places</td>\n",
       "      <td>Place</td>\n",
       "      <td>Places</td>\n",
       "      <td>Cities and Towns (Places)</td>\n",
       "      <td>US.Places</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>zip5</td>\n",
       "      <td>ZIP Code</td>\n",
       "      <td>ZIP Codes</td>\n",
       "      <td>ZIP Codes</td>\n",
       "      <td>US.ZIP5</td>\n",
       "      <td>Admin4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>csd</td>\n",
       "      <td>County Subdivision</td>\n",
       "      <td>County Subdivisions</td>\n",
       "      <td>County Subdivisions</td>\n",
       "      <td>US.CSD</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>counties</td>\n",
       "      <td>County</td>\n",
       "      <td>Counties</td>\n",
       "      <td>Counties</td>\n",
       "      <td>US.Counties</td>\n",
       "      <td>Admin3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>cbsa</td>\n",
       "      <td>CBSA</td>\n",
       "      <td>CBSAs</td>\n",
       "      <td>CBSAs</td>\n",
       "      <td>US.CBSA</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>cd</td>\n",
       "      <td>Congressional District</td>\n",
       "      <td>Congressional Districts</td>\n",
       "      <td>Congressional Districts</td>\n",
       "      <td>US.CD</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>dma</td>\n",
       "      <td>DMA</td>\n",
       "      <td>DMAs</td>\n",
       "      <td>DMAs</td>\n",
       "      <td>US.DMA</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>states</td>\n",
       "      <td>State</td>\n",
       "      <td>States</td>\n",
       "      <td>States</td>\n",
       "      <td>US.States</td>\n",
       "      <td>Admin2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>whole_usa</td>\n",
       "      <td>United States of America</td>\n",
       "      <td>United States of America</td>\n",
       "      <td>Entire Country</td>\n",
       "      <td>US.WholeUSA</td>\n",
       "      <td>Admin1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      level_name             singular_name               plural_name  \\\n",
       "0   block_groups               Block Group              Block Groups   \n",
       "1         tracts              Census Tract             Census Tracts   \n",
       "2         places                     Place                    Places   \n",
       "3           zip5                  ZIP Code                 ZIP Codes   \n",
       "4            csd        County Subdivision       County Subdivisions   \n",
       "5       counties                    County                  Counties   \n",
       "6           cbsa                      CBSA                     CBSAs   \n",
       "7             cd    Congressional District   Congressional Districts   \n",
       "8            dma                       DMA                      DMAs   \n",
       "9         states                     State                    States   \n",
       "10     whole_usa  United States of America  United States of America   \n",
       "\n",
       "                        alias        level_id admin_level  \n",
       "0                Block Groups  US.BlockGroups              \n",
       "1               Census Tracts       US.Tracts              \n",
       "2   Cities and Towns (Places)       US.Places              \n",
       "3                   ZIP Codes         US.ZIP5      Admin4  \n",
       "4         County Subdivisions          US.CSD              \n",
       "5                    Counties     US.Counties      Admin3  \n",
       "6                       CBSAs         US.CBSA              \n",
       "7     Congressional Districts           US.CD              \n",
       "8                        DMAs          US.DMA              \n",
       "9                      States       US.States      Admin2  \n",
       "10             Entire Country     US.WholeUSA      Admin1  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "usa.levels"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "66bc5566",
   "metadata": {},
   "source": [
    "## Retrive Seattle Block Groups\n",
    "\n",
    "It is not uncommon to already have the standard geography unique identifiers, if you need to retrieve those within a larger area, you can retrieve these using `standard_geography_query`. The most versatile parameter in this method is `geoquery`. You can find more explanation of the options for the `geoquery` parameter under the [geographyQuery parameter documentation](https://developers.arcgis.com/rest/geoenrichment/api-reference/standard-geography-query.htm#geographyQuery). We can start by seeing what is returned when searching for `seattle`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "7afa8a7e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>DatasetID</th>\n",
       "      <th>Hierarchy</th>\n",
       "      <th>DataLayerID</th>\n",
       "      <th>AreaID</th>\n",
       "      <th>AreaName</th>\n",
       "      <th>MajorSubdivisionName</th>\n",
       "      <th>MajorSubdivisionAbbr</th>\n",
       "      <th>MajorSubdivisionType</th>\n",
       "      <th>CountryAbbr</th>\n",
       "      <th>Score</th>\n",
       "      <th>ObjectId</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>USA_ESRI_2022</td>\n",
       "      <td>census2020</td>\n",
       "      <td>US.Places</td>\n",
       "      <td>5363000</td>\n",
       "      <td>Seattle city</td>\n",
       "      <td>Washington</td>\n",
       "      <td>WA</td>\n",
       "      <td>State</td>\n",
       "      <td>US</td>\n",
       "      <td>100</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       DatasetID   Hierarchy DataLayerID   AreaID      AreaName  \\\n",
       "0  USA_ESRI_2022  census2020   US.Places  5363000  Seattle city   \n",
       "\n",
       "  MajorSubdivisionName MajorSubdivisionAbbr MajorSubdivisionType CountryAbbr  \\\n",
       "0           Washington                   WA                State          US   \n",
       "\n",
       "   Score  ObjectId  \n",
       "0    100         1  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from arcgis.geoenrichment import standard_geography_query\n",
    "\n",
    "standard_geography_query(\"usa\", layers=\"US.Places\", geoquery=\"seattle\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bb83b4db",
   "metadata": {},
   "source": [
    "Since only one location is returned, we can use this to retrieve the block groups by populating the `sub_geography` parameteters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "094b48de",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>DatasetID</th>\n",
       "      <th>Hierarchy</th>\n",
       "      <th>DataLayerID</th>\n",
       "      <th>AreaID</th>\n",
       "      <th>AreaName</th>\n",
       "      <th>MajorSubdivisionName</th>\n",
       "      <th>MajorSubdivisionAbbr</th>\n",
       "      <th>MajorSubdivisionType</th>\n",
       "      <th>CountryAbbr</th>\n",
       "      <th>Score</th>\n",
       "      <th>ObjectId</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>USA_ESRI_2022</td>\n",
       "      <td>census2020</td>\n",
       "      <td>US.BlockGroups</td>\n",
       "      <td>530330009001</td>\n",
       "      <td>530330009.001</td>\n",
       "      <td>Washington</td>\n",
       "      <td>WA</td>\n",
       "      <td>State</td>\n",
       "      <td>US</td>\n",
       "      <td>100</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>USA_ESRI_2022</td>\n",
       "      <td>census2020</td>\n",
       "      <td>US.BlockGroups</td>\n",
       "      <td>530330009002</td>\n",
       "      <td>530330009.002</td>\n",
       "      <td>Washington</td>\n",
       "      <td>WA</td>\n",
       "      <td>State</td>\n",
       "      <td>US</td>\n",
       "      <td>100</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>USA_ESRI_2022</td>\n",
       "      <td>census2020</td>\n",
       "      <td>US.BlockGroups</td>\n",
       "      <td>530330010001</td>\n",
       "      <td>530330010.001</td>\n",
       "      <td>Washington</td>\n",
       "      <td>WA</td>\n",
       "      <td>State</td>\n",
       "      <td>US</td>\n",
       "      <td>100</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>USA_ESRI_2022</td>\n",
       "      <td>census2020</td>\n",
       "      <td>US.BlockGroups</td>\n",
       "      <td>530330010002</td>\n",
       "      <td>530330010.002</td>\n",
       "      <td>Washington</td>\n",
       "      <td>WA</td>\n",
       "      <td>State</td>\n",
       "      <td>US</td>\n",
       "      <td>100</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>USA_ESRI_2022</td>\n",
       "      <td>census2020</td>\n",
       "      <td>US.BlockGroups</td>\n",
       "      <td>530330011002</td>\n",
       "      <td>530330011.002</td>\n",
       "      <td>Washington</td>\n",
       "      <td>WA</td>\n",
       "      <td>State</td>\n",
       "      <td>US</td>\n",
       "      <td>100</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       DatasetID   Hierarchy     DataLayerID        AreaID       AreaName  \\\n",
       "0  USA_ESRI_2022  census2020  US.BlockGroups  530330009001  530330009.001   \n",
       "1  USA_ESRI_2022  census2020  US.BlockGroups  530330009002  530330009.002   \n",
       "2  USA_ESRI_2022  census2020  US.BlockGroups  530330010001  530330010.001   \n",
       "3  USA_ESRI_2022  census2020  US.BlockGroups  530330010002  530330010.002   \n",
       "4  USA_ESRI_2022  census2020  US.BlockGroups  530330011002  530330011.002   \n",
       "\n",
       "  MajorSubdivisionName MajorSubdivisionAbbr MajorSubdivisionType CountryAbbr  \\\n",
       "0           Washington                   WA                State          US   \n",
       "1           Washington                   WA                State          US   \n",
       "2           Washington                   WA                State          US   \n",
       "3           Washington                   WA                State          US   \n",
       "4           Washington                   WA                State          US   \n",
       "\n",
       "   Score  ObjectId  \n",
       "0    100         1  \n",
       "1    100         2  \n",
       "2    100         3  \n",
       "3    100         4  \n",
       "4    100         5  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bg_df = standard_geography_query(\n",
    "    \"usa\",\n",
    "    layers=\"US.Places\",\n",
    "    geoquery=\"seattle\",\n",
    "    sub_geography_layer=\"US.BlockGroups\",\n",
    "    return_sub_geography=True,\n",
    ")\n",
    "\n",
    "bg_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af1551a5",
   "metadata": {},
   "source": [
    "## Enrich\n",
    "\n",
    "Now, we can use the retrieved block groups as input into the `enrich` method to acheive the same results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "a660c3f4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>std_geography_level</th>\n",
       "      <th>std_geography_name</th>\n",
       "      <th>std_geography_id</th>\n",
       "      <th>source_country</th>\n",
       "      <th>aggregation_method</th>\n",
       "      <th>population_to_polygon_size_rating</th>\n",
       "      <th>apportionment_confidence</th>\n",
       "      <th>has_data</th>\n",
       "      <th>nohs_cy</th>\n",
       "      <th>somehs_cy</th>\n",
       "      <th>...</th>\n",
       "      <th>val300k_cy</th>\n",
       "      <th>val400k_cy</th>\n",
       "      <th>val500k_cy</th>\n",
       "      <th>val750k_cy</th>\n",
       "      <th>val1m_cy</th>\n",
       "      <th>medval_cy</th>\n",
       "      <th>avgval_cy</th>\n",
       "      <th>valbase_cy</th>\n",
       "      <th>wlthindxcy</th>\n",
       "      <th>SHAPE</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>US.BlockGroups</td>\n",
       "      <td>530330009.001</td>\n",
       "      <td>530330009001</td>\n",
       "      <td>USA</td>\n",
       "      <td>Query:US.BlockGroups</td>\n",
       "      <td>2.191</td>\n",
       "      <td>2.576</td>\n",
       "      <td>1</td>\n",
       "      <td>13.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>...</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>68.0</td>\n",
       "      <td>103.0</td>\n",
       "      <td>176.0</td>\n",
       "      <td>1022727.0</td>\n",
       "      <td>1040915.0</td>\n",
       "      <td>366.0</td>\n",
       "      <td>357.0</td>\n",
       "      <td>{\"rings\": [[[-122.280028912264, 47.71915262818...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>US.BlockGroups</td>\n",
       "      <td>530330009.002</td>\n",
       "      <td>530330009002</td>\n",
       "      <td>USA</td>\n",
       "      <td>Query:US.BlockGroups</td>\n",
       "      <td>2.191</td>\n",
       "      <td>2.576</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>29.0</td>\n",
       "      <td>41.0</td>\n",
       "      <td>85.0</td>\n",
       "      <td>1519784.0</td>\n",
       "      <td>1444495.0</td>\n",
       "      <td>327.0</td>\n",
       "      <td>344.0</td>\n",
       "      <td>{\"rings\": [[[-122.27725891127136, 47.716030628...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>US.BlockGroups</td>\n",
       "      <td>530330010.001</td>\n",
       "      <td>530330010001</td>\n",
       "      <td>USA</td>\n",
       "      <td>Query:US.BlockGroups</td>\n",
       "      <td>2.191</td>\n",
       "      <td>2.576</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>29.0</td>\n",
       "      <td>...</td>\n",
       "      <td>10.0</td>\n",
       "      <td>31.0</td>\n",
       "      <td>137.0</td>\n",
       "      <td>46.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>634124.0</td>\n",
       "      <td>655349.0</td>\n",
       "      <td>229.0</td>\n",
       "      <td>93.0</td>\n",
       "      <td>{\"rings\": [[[-122.29380691276572, 47.711973626...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>US.BlockGroups</td>\n",
       "      <td>530330010.002</td>\n",
       "      <td>530330010002</td>\n",
       "      <td>USA</td>\n",
       "      <td>Query:US.BlockGroups</td>\n",
       "      <td>2.191</td>\n",
       "      <td>2.576</td>\n",
       "      <td>1</td>\n",
       "      <td>5.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>...</td>\n",
       "      <td>5.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>114.0</td>\n",
       "      <td>76.0</td>\n",
       "      <td>74.0</td>\n",
       "      <td>789474.0</td>\n",
       "      <td>857270.0</td>\n",
       "      <td>282.0</td>\n",
       "      <td>255.0</td>\n",
       "      <td>{\"rings\": [[[-122.2908289113606, 47.7067966266...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>US.BlockGroups</td>\n",
       "      <td>530330011.002</td>\n",
       "      <td>530330011002</td>\n",
       "      <td>USA</td>\n",
       "      <td>Query:US.BlockGroups</td>\n",
       "      <td>2.191</td>\n",
       "      <td>2.576</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>...</td>\n",
       "      <td>11.0</td>\n",
       "      <td>41.0</td>\n",
       "      <td>219.0</td>\n",
       "      <td>130.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>684361.0</td>\n",
       "      <td>738695.0</td>\n",
       "      <td>429.0</td>\n",
       "      <td>207.0</td>\n",
       "      <td>{\"rings\": [[[-122.30622891353259, 47.706644624...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 106 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "  std_geography_level std_geography_name std_geography_id source_country  \\\n",
       "0      US.BlockGroups      530330009.001     530330009001            USA   \n",
       "1      US.BlockGroups      530330009.002     530330009002            USA   \n",
       "2      US.BlockGroups      530330010.001     530330010001            USA   \n",
       "3      US.BlockGroups      530330010.002     530330010002            USA   \n",
       "4      US.BlockGroups      530330011.002     530330011002            USA   \n",
       "\n",
       "     aggregation_method  population_to_polygon_size_rating  \\\n",
       "0  Query:US.BlockGroups                              2.191   \n",
       "1  Query:US.BlockGroups                              2.191   \n",
       "2  Query:US.BlockGroups                              2.191   \n",
       "3  Query:US.BlockGroups                              2.191   \n",
       "4  Query:US.BlockGroups                              2.191   \n",
       "\n",
       "   apportionment_confidence  has_data  nohs_cy  somehs_cy  ...  val300k_cy  \\\n",
       "0                     2.576         1     13.0        7.0  ...         2.0   \n",
       "1                     2.576         1      0.0        0.0  ...         0.0   \n",
       "2                     2.576         1      0.0       29.0  ...        10.0   \n",
       "3                     2.576         1      5.0        9.0  ...         5.0   \n",
       "4                     2.576         1      0.0        8.0  ...        11.0   \n",
       "\n",
       "   val400k_cy  val500k_cy  val750k_cy  val1m_cy  medval_cy  avgval_cy  \\\n",
       "0         2.0        68.0       103.0     176.0  1022727.0  1040915.0   \n",
       "1         3.0        29.0        41.0      85.0  1519784.0  1444495.0   \n",
       "2        31.0       137.0        46.0       4.0   634124.0   655349.0   \n",
       "3        10.0       114.0        76.0      74.0   789474.0   857270.0   \n",
       "4        41.0       219.0       130.0       9.0   684361.0   738695.0   \n",
       "\n",
       "   valbase_cy  wlthindxcy                                              SHAPE  \n",
       "0       366.0       357.0  {\"rings\": [[[-122.280028912264, 47.71915262818...  \n",
       "1       327.0       344.0  {\"rings\": [[[-122.27725891127136, 47.716030628...  \n",
       "2       229.0        93.0  {\"rings\": [[[-122.29380691276572, 47.711973626...  \n",
       "3       282.0       255.0  {\"rings\": [[[-122.2908289113606, 47.7067966266...  \n",
       "4       429.0       207.0  {\"rings\": [[[-122.30622891353259, 47.706644624...  \n",
       "\n",
       "[5 rows x 106 columns]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "enrich_df = usa.enrich(\n",
    "    bg_df,\n",
    "    enrich_variables=enrich_vars,\n",
    "    standard_geography_level=\"block_groups\",\n",
    "    standard_geography_id_column=\"AreaID\",\n",
    ")\n",
    "\n",
    "enrich_df.head()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "f58bfe3e",
   "metadata": {},
   "source": [
    "## Calculate Variance\n",
    "\n",
    "Variation can now be calculated for the retrieved variables to identify those with exceedingly high variance. Analysis can be used for feature selection or feature reduction to address covariance between variables and perform modeling."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ad3bd678",
   "metadata": {},
   "source": [
    "Just as in the previous notebook, we can evaluate the variance and select the top variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "4fe84850",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>alias</th>\n",
       "      <th>variance</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>aggdi_cy</th>\n",
       "      <td>AGGDI_CY</td>\n",
       "      <td>2022 Aggregate Disposable Income</td>\n",
       "      <td>9.773137e+14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>aggdi_cy</th>\n",
       "      <td>AGGDI_CY</td>\n",
       "      <td>2022 Aggregate Disposable Income</td>\n",
       "      <td>9.773137e+14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>agghinc_cy</th>\n",
       "      <td>AGGHINC_CY</td>\n",
       "      <td>2022 Aggregate HH Income</td>\n",
       "      <td>2.220839e+15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>agghinc_cy</th>\n",
       "      <td>AGGHINC_CY</td>\n",
       "      <td>2022 Aggregate HH Income</td>\n",
       "      <td>2.220839e+15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>agghinc_cy</th>\n",
       "      <td>AGGHINC_CY</td>\n",
       "      <td>2022 Aggregate HH Income</td>\n",
       "      <td>2.220839e+15</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  name                             alias      variance\n",
       "aggdi_cy      AGGDI_CY  2022 Aggregate Disposable Income  9.773137e+14\n",
       "aggdi_cy      AGGDI_CY  2022 Aggregate Disposable Income  9.773137e+14\n",
       "agghinc_cy  AGGHINC_CY          2022 Aggregate HH Income  2.220839e+15\n",
       "agghinc_cy  AGGHINC_CY          2022 Aggregate HH Income  2.220839e+15\n",
       "agghinc_cy  AGGHINC_CY          2022 Aggregate HH Income  2.220839e+15"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# get just the enrich value columns\n",
    "enrich_cols = [\n",
    "    c for c in enrich_df if c in usa.enrich_variables.name.str.lower().values\n",
    "]\n",
    "enrich_df = enrich_df.set_index(\"std_geography_id\").loc[:, enrich_cols]\n",
    "\n",
    "# get top 20 highest variance columns\n",
    "top20 = enrich_df.var(ddof=0).sort_values(ascending=False).iloc[:20]\n",
    "top20.name = \"variance\"\n",
    "\n",
    "# add human readable names\n",
    "ev = usa.enrich_variables\n",
    "ev.index = ev.name.str.lower()\n",
    "top20_df = ev.join(top20, how=\"right\").loc[:, [\"name\", \"alias\", \"variance\"]]\n",
    "\n",
    "top20_df.head()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "1acb4644",
   "metadata": {},
   "source": [
    "## Continuing Analysis\n",
    "\n",
    "From here, a variety of techniques can be used, but with so many income and net worth variables, before subsequent modeling steps, covariance needs to be addressed. Using the GeoEnrichment dramatically streamlines getting to this point. It provides extremely easy access to thousands of demographic variables for modeling and analysis directly in Python, making it easy to integrate with data engineering pipelines."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.9.11 64-bit",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.11 [MSC v.1931 64 bit (AMD64)]"
  },
  "vscode": {
   "interpreter": {
    "hash": "07c13af76457a6f4e5d5b34d0e1bc42b2e017343b05b916c8871f483eed35ce6"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
